Annotation detection and anchoring on ink notes

ABSTRACT

Systems and methods for detecting annotation digital ink strokes and further associating annotation digital ink strokes with word digital ink strokes are presented. Ink strokes are captured on a writing surface and then classified as words or annotations. Annotations are then anchored to corresponding words. When words are relocated or edited on the writing surface, the anchored annotations are also relocated and may even be reshaped according to the changes in the anchored words.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/275,398, filed on Dec. 29, 2005, and entitled “Annotation Detectionand Anchoring on Ink Notes”, the entirety of which is incorporatedherein by reference.

BACKGROUND

Computer users are accustomed to using a mouse and keyboard as a way ofinteracting with a personal computer. While personal computers provide anumber of advantages over printed documents, users often continue toperform certain functions using printed paper. Some of these functionsinclude reading handwritten documents. In the case of annotations, theprinted document assumes a greater significance because of theannotations placed on it by the user. One of the difficulties, however,with having a hand-written document is a later need to have the contententered back into the electronic form of the document. This requires theoriginal user or another user to wade through the handwritten contentand enter them into a personal computer. In some cases, a user will scanin the handwritten content and the original text, thereby creating a newdocument. These multiple steps make the interaction between the printeddocument and the electronic version of the document difficult to handleon a repeated basis. Further, scanned-in images are frequentlynon-modifiable. There may be no way to separate the handwritten contentfrom the original text. This makes using the handwritten contentdifficult. Accordingly, an improved way of handling handwritten contentis needed.

Tablet PC's are increasingly being used to capture and displayhandwritten content. In the context of taking notes during a meeting,for example, a Tablet PC presents less of a barrier between individualsthan does a laptop computer. Tablet PC's also allow for capturing ofhandwritten notes so that Tablet PC users are able to take notes intheir own handwriting, in an unobtrusive manner similar to takinghandwritten notes with a pen and paper.

Tablet PC's capture and display digital ink. The term “digital ink”refers to one or more strokes that are recorded from a pointing device,such as a mouse, a stylus/pen on a digitizer tablet, or a stylus/pen ona display screen integrated with a digitizer tablet (e.g., atouch-sensitive display screen). As used herein, the term “ink” isshorthand for digital ink.

Documents displayed on writing surfaces of tablet PCs may be annotatedby a user. Correspondingly, annotations such as underline and blobs areoften incorporated within the document. For example, a user mayunderline a few words in a document to emphasize the words underlined.Thus, extraction of handwritten annotations plays an important role indocument processing.

SUMMARY

An input device is used to enter a plurality of digital ink strokes on awriting surface, such as a Tablet PC screen. Digital ink strokes arecaptured and may then be classified. The digital ink strokes may beclassified as word ink strokes and annotation ink strokes. Annotationink strokes are anchored to corresponding word ink strokes. When wordink strokes are relocated on the writing surface, the anchoredannotation ink strokes are also relocated.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the clamed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF DRAWINGS

The foregoing summary of the invention, as well as the followingdetailed description of preferred embodiments, is better understood whenread in conjunction with the accompanying drawings, which are includedby way of example, and not by way of limitation with regard to theclaimed invention.

FIG. 1 is a functional block diagram of an example of a conventionalgeneral-purpose digital computing environment that can be used toimplement various aspects of the present invention.

FIG. 2 shows an exemplary document that may be displayed on a computerdevice writing surface, in accordance with an embodiment of theinvention.

FIG. 3 illustrates a system that maybe used to classify and anchoringannotations. Digital ink stroke data may be delivered to one or moreclassification and anchoring engines, in accordance with an embodimentof the invention.

FIG. 4 shows a document having modified annotation positions, inaccordance with an embodiment of the invention.

FIG. 5 illustrates a method of associating an annotation with one ormore words represented by a plurality of digital ink strokes, inaccordance with an embodiment of the invention.

DETAILED DESCRIPTION

Aspects of the present invention relate to classifying annotationdigital ink strokes and anchoring the annotation digital ink strokeswith word digital ink strokes. FIG. 1 is a functional block diagram ofan example of a conventional general-purpose Tablet PC digital computingenvironment that can be used to process a writing input from a Pen(e.g., stylus 166). As used herein, the term Tablet PC refers to, by wayof example, without limitation, a full-function “MICROSOFT” “WINDOWS”operating system-based personal computer incorporating variousconvenient and intuitive aspects of pencil and paper into a user'sinteraction with a PC. In FIG. 1, a computer 100 includes a processingunit 110, a system memory 120, and a system bus 130 that couples varioussystem components including the system memory to the processing unit110. The system bus 130 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. The system memory120 includes read only memory (ROM) 140 and random access memory (RAM)150.

A basic input/output system 160 (BIOS), containing the basic routinesthat help to transfer information between elements within the computer100, such as during start-up, is stored in the ROM 140. The computer 100also includes a hard disk drive 170 for reading from and writing to ahard disk (not shown), a magnetic disk drive 180 for reading from orwriting to a removable magnetic disk 190, and an optical disk drive 191for reading from or writing to a removable optical disk 192 such as a CDROM or other optical media. The hard disk drive 170, magnetic disk drive180, and optical disk drive 191 are connected to the system bus 130 by ahard disk drive interface 192, a magnetic disk drive interface 193, andan optical disk drive interface 194, respectively. The drives and theirassociated computer-readable media provide nonvolatile storage ofcomputer readable instructions, data structures, program modules andother data for the personal computer 100. It will be appreciated bythose skilled in the art that other types of computer readable mediathat can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, Bernoullicartridges, random access memories (RAMs), read only memories (ROMs),and the like, may also be used in the example operating environment.

A number of program modules can be stored on the hard disk drive 170,magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including anoperating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a microphone, joystick, game pad, satellite dish, scanner orthe like. These and other input devices are often connected to theprocessing unit 110 through a serial port interface 106 that is coupledto the system bus, but may be connected by other interfaces, such as aparallel port, game port or a universal serial bus (USB). Further still,these devices may be coupled directly to the system bus 130 via anappropriate interface (not shown). A monitor 107 or other type ofdisplay device is also connected to the system bus 130 via an interface,such as a video adapter 108. In addition to the monitor, personalcomputers typically include other peripheral output devices (not shown),such as speakers and printers.

In an embodiment, a writing surface, such as a pen digitizer 165, andaccompanying pen or stylus 166 are provided in order to digitallycapture freehand input to provide a writing input (e.g., a handwrittendocument) for processing unit 110. Although a direct connection betweenthe pen digitizer 165 and the serial port is shown, in practice, the pendigitizer 165 may be coupled to the processing unit 110 directly, via aparallel port or other interface and the system bus 130 as known in theart. Furthermore, although the digitizer 165 is shown apart from themonitor 107, it is preferred that the usable input area of the digitizer165 be co-extensive with the display area of the monitor 107. Furtherstill, the digitizer 165 may be integrated in the monitor 107, or mayexist as a separate device overlaying or otherwise appended to themonitor 107.

The computer 100 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer109. The remote computer 109 can be a server, a router, a network PC, apeer device or other common network node, and typically includes many orall of the elements described above relative to the computer 100,although only a memory storage device 111 has been illustrated inFIG. 1. The logical connections depicted in FIG. 1 include a local areanetwork (LAN) 112 and a wide area network (WAN) 113. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

When used in a LAN networking environment, the computer 100 is connectedto the local network 112 through a network interface or adapter 114.When used in a WAN networking environment, the personal computer 100typically includes a modem 115 or other means for establishing acommunications over the wide area network 113, such as the Internet. Themodem 115, which may be internal or external, is connected to the systembus 130 via the serial port interface 106. In a networked environment,program modules depicted relative to the personal computer 100, orportions thereof, may be stored in the remote memory storage device.

Aspects of the invention relate to annotation detection and anchoring.An annotation may be in the form of an underline, blob or any otherannotation, e.g. callout, that a user would add to a document. Anunderline, for example, refers to a stroke or a series of strokes drawnunder lines of a document. An underline annotates the words above theunderline. A blob consists of one or more digital ink strokes that forma closed pattern. A blob may encloses words, table, an image or otherelements of a document. A blob annotates the words or other elementssurrounded by the digital ink strokes(s).

FIG. 2 shows an exemplary document 200 that may be displayed on acomputer device writing surface, such as a Tablet PC writing surface. Increating document 200, a user may use a pen to create digital inkstrokes. The digital ink strokes are captured by the computer device andthen displayed to the user. FIG. 2 shows that several digital inkstrokes form words. For example, a group of digital ink strokes 202 formthe word “click.” Other digital ink strokes form annotations. Digitalink strokes 204 form a blob annotation that surrounds digital inkstrokes that form the words “inbox” 204 a and “folder” 204 b. Digitalink strokes 206 form an underline annotation that is positioned belowseveral word digital ink strokes that form the words “select” 206 a,“rules” 206 b, “and” 206 c, “alerts” 206 d″ and “from” 206 e.

FIG. 3 illustrates a system that maybe used to classify and anchorannotations. Digital ink stroke data may be delivered to one or moreclassification and anchoring engines. Each digital ink stroke may bestored as one or more ink packets, in which each ink packet may containcoordinates (x, y) corresponding to the position of the pointing device.For example, a user may move a pen along a touch-sensitive displayscreen of a computer system so as to draw a line or curve, and thecomputer system may sample the coordinates (x, y) along the trajectoryof the pen tip position over time (or on any other interval as known inthe art) as the user moves the pen. These coordinates represent pointsalong the curve or line and are stored as ink packets.

An underline classification and anchoring engine 304 may be configuredto receive digital ink stroke data 302 and determine which ink strokesrepresent underline annotations. Underline classification and anchoringengine 304 may utilize a processor, computer-executable instructionsand/or a rules engine configured to compare attributes of the inkstrokes to underline annotation criteria 306. For example, underlineclassification and anchoring engine 304 may classify one or more inkstrokes as representing an underline if the ink strokes are beneath andadjacent to writing lines. One skilled in the art will appreciate thatvarious other methods exist for classifying digital ink strokes as worddigital ink strokes. Other criteria may also be used to classify inkstrokes as underline ink strokes. For example, underline classificationand anchoring engine 304 may be configured to only consider ink strokesthat are drawing ink strokes. Drawing ink strokes are those that do nothave attributes of word ink strokes and are part of graphical objects.

Other criteria may include classifying an ink stroke as an underline inkstroke when the ink stroke has a relatively large width to height ratioor when the ink stroke overlaps a word in a horizontal direction. Oneskilled in the art will appreciate that additional, alternative andcombinations of criteria may be used to classify an ink stroke as anunderline ink stroke. Underline may also be composed of multiplestrokes. Dynamic programming may be used to identify strokes that arepart of the same underline annotation and determine when the search forstrokes that are part of the same underline should end. This may includeanalyzing the features described above. It may then be determined if inkstrokes close to one another form a single underline, for example,considering the proximity of the ink strokes.

Underline classification and anchoring engine 304 may also be configuredto anchor underline annotations to words or other document elements.Anchoring may include associating one or more underline ink strokes withone or more word or other ink strokes. Underline data 308 may include anink stroke identification that uniquely identifies the ink stroke andanchoring data that uniquely identifies the word ink strokes or otherelements to which the annotation ink stroke is anchored.

Once annotation ink strokes are properly anchored, the document may berearranged without loosing the annotations. For example, if a userunderlines a segment of text and then moves the text segment to anotherpart of the document, the annotation can be recreated at the newlocation. FIG. 4 shows a document 400 that is a modified version ofdocument 200 (shown in FIG. 2). Words 206 a, 206 b, 206 d, 206 e and 204a have been moved from the positions they had in document 200. Themovement may result from the user selecting one or more words 206 a, 206b, 206 d, 206 e and 204 a with a pointing device and dragging words 206a, 206 b, 206 d, 206 e and 204 a to their new locations. Underlineannotation 402 was anchored to words 206 a and 206 b and underlineannotation 404 was anchored to words 206 d and 206 e. A single underlineannotation may be broken into two or more underline annotations. Forexample, in FIG. 2 underline annotation 206 is under words 206 a-206 e.When word “and” 206 c is removed in FIG. 4 underline annotations 402 and404 are formed under the remaining words. Similarly, blob annotation 204is formed around words 204 a and 204 b in FIG. 2. When the word “folder”204 b is removed in FIG. 4, blob annotation 406 is formed only aroundthe word “inbox” 204 a. Of course, the size of a blob annotation may bereduced or increased to fit around a given collection of words.

The system shown in FIG. 3 also includes a blob classification andanchoring engine 310. Blob classification and anchoring engine 310 maybe configured to operate in a manner similar to that of underlineclassification and anchoring engine 304. An exemplary criterion used toclassify an ink stroke as a blob includes determining whether one ormore ink strokes form a closed pattern. Of course, a “closed pattern” asused herein is meant to encompass a substantially enclosed pattern.Another criterion may include whether or not the closed pattern encloseswords or other document elements. Blob classification and anchoringengine 308 may produce blob data 310.

One skilled in the art will appreciate that aspects of the invention maybe used to classify and anchor other types of annotations, such asvertical brackets, call outs and highlighting. A miscellaneousclassification and anchoring engine 314 is included in the system shownin FIG. 3 to show that the system may be adapted to classify and anchorother types of annotations and may produce annotation data 316. Whenmiscellaneous classification and anchoring engine 314 is used toclassify multiple types of annotations, annotation data 316 may includea field that identifies the type of annotation. Underline data 308 andblob data 312 may also include fields that identify the type ofannotation.

FIG. 5 illustrates a method of associating an annotation with one ormore words represented by a plurality of digital ink strokes. First, instep 502 a plurality of digital ink strokes are received. The digitalink strokes may be filtered so that they consist only of drawing inksstrokes and/or are not words, images or other document elements. Next,in step 504 at least one digital ink stroke is classified as anannotation. They classification may use the criteria described above andmay result in the digital ink stroke(s) being classified as anunderline, blob or other type of annotation. After annotation digitalink strokes are identified, they are anchored to one or more digital inkstrokes that represent one or more words in step 506. Next, it isdetermined whether a command to reposition word digital ink strokes on awriting surface are received in step 508. When a command is notreceived, in step 510 the process waits a predetermined time periodbefore performing step 508 again. When a command is received, in step512, the word digital ink strokes and anchored annotation digital inkstrokes are repositioned.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

We claim:
 1. A computer-implemented method comprising: receiving digitalink strokes; classifying at least some of the digital ink strokes asword digital ink strokes; classifying at least some of the digital inkstrokes as annotation digital ink strokes; anchoring the annotationdigital ink strokes to the word digital ink strokes to representannotated words; and breaking the annotation digital ink strokes intotwo separate annotations responsive to receiving a command to delete aportion of the annotated words that are between other portions of theannotated words, each of the two separate annotations maintaining anassociated previously anchored portion of the annotation digital inkstrokes associated with annotated words that were not deleted.
 2. Themethod of claim 1, further comprising classifying a first of one or moreof the digital ink strokes as a first of the two separate annotationsand a second of one or more of the digital ink strokes as a second ofthe two separate annotations.
 3. The method of claim 1, furthercomprising resizing the first and the second of the two separateannotations based on a size of the annotated words that were not deletedthat are associated with the first and the second of the two separateannotations.
 4. The method of claim 1, wherein the annotation comprisesan underline.
 5. The method of claim 1, wherein the annotation comprisesa blob.
 6. The method of claim 1, the method further comprisingreceiving a command to reposition, on a writing surface, correspondinganchored word digital ink strokes associated with a first of the twoseparate annotations.
 7. The method of claim 6, wherein in response toreceiving the command: repositioning the corresponding anchored worddigital ink strokes; repositioning the first of one or more of thedigital ink strokes to maintain correlation with the correspondinganchored word digital ink strokes; and repositioning at least some ofthe word digital ink strokes to accommodate the repositioning of thecorresponding anchored word digital ink strokes.
 8. The method of claim7, the method further comprising reshaping the corresponding anchoredword digital ink strokes and the first of one or more digital inkstrokes to accommodate the repositioning.
 9. The method of claim 6 themethod further comprising: receiving a command to reposition, on awriting surface, corresponding anchored word digital ink strokesassociated with a second of the two separate annotations; and inresponse to receiving the command: repositioning the correspondinganchored word digital ink strokes; repositioning the second of one ormore of the digital ink strokes to maintain correlation with thecorresponding anchored word digital ink strokes; and repositioning atleast some of the word digital ink strokes to accommodate therepositioning of the corresponding anchored word digital ink strokes.10. The method of claim 9, the method further comprising reshaping thecorresponding anchored word digital ink strokes and the second of one ormore of the digital ink strokes to accommodate the repositioning.
 11. Acomputer-implemented method of associating an annotation with a sequenceof adjacent words represented by a plurality of digital ink strokes on awriting surface, the method comprising: classifying at least one digitalink stroke as an underline annotation; anchoring the at least onedigital ink stroke to the sequence of adjacent words represented by theplurality of digital ink strokes; and breaking the underline annotationinto at least two separate underline annotations responsive to a userinput that deletes at least one portion of the sequence of adjacentwords being between other portions of the sequence of adjacent words,each of the at least two separate underline annotations maintaining anassociated previously anchored portion of the sequence of adjacent wordsthat were not deleted.
 12. The method of claim 11, further comprising:classifying at least one additional digital ink stroke as an underlineannotation for each of the at least two separate underline annotations;and responsive to a user input for repositioning an associatedpreviously anchored portion of the sequence of adjacent words associatedwith a first of the at least two separate underline annotations,repositioning the associated previously anchored portion of the sequenceof adjacent words associated with the first of the at least two separateunderline annotations to a different position on the writing surface.13. The method of claim 11, wherein the classifying comprisesdetermining if the at least one ink stroke overlaps a word in ahorizontal direction.
 14. The method of claim 11, wherein the anchoringcomprises associating ink stroke information that uniquely identifies,within a document, the at least one ink stroke to anchoring dataassociated with the sequence of adjacent words.
 15. The method of claim11, wherein the classifying comprises determining if the at least oneink stroke is beneath and adjacent to writing lines.
 16. The method ofclaim 15, wherein the classifying comprises determining if the at leastone ink stroke is a drawing stroke.
 17. The method of claim 11, furthercomprising: responsive to a user input that deletes a beginning or anending portion of the sequence of adjacent words, reshaping a remainingportion of the sequence to include the underline annotation thatcorresponds to deletion of the portion of the sequence.
 18. A systemcomprising: one or more processors; memory; and programming instructionsstored on the memory that, when executed by the one or more processors,configure the one or more processors to perform operations comprising:receiving digital ink strokes; classifying at least some of the digitalink strokes as word digital ink strokes; classifying at least some ofthe digital ink strokes as annotation digital ink strokes; anchoring theannotation digital ink strokes to the word digital ink strokes torepresent annotated words; and breaking the annotation digital inkstrokes into two separate annotations responsive to receiving a commandto delete a portion of the annotated words that are between otherportions of the annotated words, each of the two separate annotationsmaintaining an associated previously anchored portion of the annotationdigital ink strokes associated with annotated words that were notdeleted.
 19. The system of claim 18, further comprising classifying afirst of one or more of the digital ink strokes as a first of the twoseparate annotations and a second of one or more of the digital inkstrokes as a second of the two separate annotations.
 20. The system ofclaim 18, further comprising resizing the first and the second of thetwo separate annotations based on a size of the annotated words thatwere not deleted that are associated with the first and the second ofthe two separate annotations.